{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.9 Adadelta算法\n",
    "“长江后浪推前浪，一浪更比一浪高”。RMSProp算法对Adagrad算法在学习率上进行了改进。同时期，其他的研究者还提出了不同的改进算法。其中比较典型的就是Adadelta算法啦。在优化算法的发展史上留下了值得炫耀的一笔，本节咱们就来介绍下。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.9.1 基本原理\n",
    "\n",
    "从时间线上来看，Adagrad算法是由John Duchi、Eliza Singer和Yoshua Bengio在2011年提出的。RMSProp算法是由Geoffrey Hinton在2012年提出的。Adadelta算法是由Matthew D. Zeiler在2012年提出的。基本上是脚前脚后，同一个时期。\n",
    "\n",
    "Adadelta是一种自适应学习率的方法，用于神经网络的训练过程中。它的基本思想是避免使用手动调整学习率的方法来控制训练过程，而是自动调整学习率，使得训练过程更加顺畅。\n",
    "\n",
    "Adadelta算法主要由两部分组成：梯度积分和更新规则。\n",
    "\n",
    "梯度积分：在每一个时刻，我们可以计算出每一层的梯度。梯度积分即是对这些梯度进行累加，并记录下来。\n",
    "\n",
    "更新规则：我们通过使用梯度积分来更新每一层的权重。我们使用如下公式来计算权重的更新量：\n",
    "\n",
    "$$\\Delta w = -\\frac{\\sqrt{E[g^2] + \\epsilon}}{\\sqrt{E[\\Delta w^2] + \\epsilon}} \\cdot g$$\n",
    "\n",
    "其中，$\\Delta w$ 表示权重的更新量，$g$ 表示当前梯度，$E[g^2]$ 表示梯度积分，$E[\\Delta w^2]$ 表示权重更新量的积分，$\\epsilon$ 是一个很小的正数，用于防止分母为 $0$ 的情况。\n",
    "\n",
    "我们使用这个公式来计算权重的更新量，并更新权重。这样，我们就实现了使用自适应学习率来训练神经网络的目的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.9.2 优缺点\n",
    "\n",
    "从优点来说，Adadelta算法自动调整学习率，使得训练过程更加顺畅。它不需要手动调整学习率，可以省去调参的时间，而且能够在训练过程中避免出现“饱和”现象，使得训练更加稳定。\n",
    "\n",
    "从缺点来看，它可能会收敛得比较慢，因为它不会显式地调整学习率。此外，它需要维护梯度和权重更新量的积分，可能会增加空间复杂度。\n",
    "\n",
    "### 6.9.3 代码示例\n",
    "\n",
    "在下面这个例子中，我们将演示用一个简单的神经网络来拟合 $y = x^2$ 这个函数，选取均方误差作为损失函数，使用Adadelta算法来优化权重。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAj8klEQVR4nO3deXhV5bn38e+dnZ0RSAiEMYEgoDKogBERZ5zQOnRQX61TWy21tVVb39NaT1tP7Xl7ejrY1g5aKlY9jnWoeqwTtTjgAAREQECZJxmCYQ4h0/3+sRc2xoRsYCcrWfl9rmtfWftZz977Xi75ZeXZz1rL3B0REYmutLALEBGR1qWgFxGJOAW9iEjEKehFRCJOQS8iEnHpYRfQlJ49e3pJSUnYZYiIdBizZ8/e7O6FTa1rl0FfUlJCWVlZ2GWIiHQYZraquXUauhERiTgFvYhIxCnoRUQiTkEvIhJxCnoRkYhT0IuIRFyLQW9mWWY208zeNbP3zOzHTfT5kpmVm9nc4HFNg3VXmdmS4HFVqjdARET2LZl59HuACe6+08ziwHQze97d327U71F3/2bDBjMrAG4FSgEHZpvZM+6+JRXFN1RVU8f/vLWKkf3zOG5wj1S/vYhIh9XiEb0n7AyexoNHshexPwuY6u4VQbhPBSYeUKUtiKUZd09fzp2vLmuNtxcR6bCSGqM3s5iZzQU2kQjuGU10+4KZzTOzx82sOGjrD6xp0Gdt0NbUZ0wyszIzKysvL09+CwLxWBqXHzuQ1z4oZ+mmnS2/QESkk0gq6N29zt1HAUXAWDMb2ajL/wIl7n4kiaP2+/a3EHef7O6l7l5aWNjk5Rpa9MVjB5CRnsZ9b648oNeLiETRfs26cfetwDQaDb+4+0fuvid4ejdwdLC8Dihu0LUoaGsVPbpkcv5R/Xhizlq27a5prY8REelQkpl1U2hm+cFyNnAGsLhRn74Nnp4PLAqWXwTONLPuZtYdODNoazVfGl9CZXUdj5WtabmziEgnkMwRfV9gmpnNA2aRGKN/1sxuM7Pzgz7XB1Mv3wWuB74E4O4VwE+C180CbgvaWs3I/nmMLSngvrdWUlevG5+LiJh7+wvD0tJSP5jLFD83fz3feHAOk684mjNH9ElhZSIi7ZOZzXb30qbWRfLM2DOH96ZfXhZ/eWNl2KWIiIQukkGfHkvjiuNKeGv5Ryxavz3sckREQhXJoAe4dGwxWfE07tVRvYh0cpEN+vycDD4/poin5q6jYld12OWIiIQmskEP8OXxJeyprefhmavDLkVEJDSRDvqhvbty4tCe/M9bq6ipqw+7HBGRUEQ66AG+fHwJG7ZX8fyCDWGXIiISisgH/SmH9mJQz1zumb4i7FJEREIR+aBPSzO+NL6EuWu2MntVyi+DLyLS7kU+6AEuPLqIblnpTJm+POxSRETaXKcI+tzMdC49dgAvLNjAmorKsMsREWlTnSLoAa46rgQz07XqRaTT6TRB3y8/m3OO6Mujs9awc09t2OWIiLSZThP0AFefMIgde2r56yxdq15EOo9OFfSjivM5emB3/vLmCl2rXkQ6jU4V9ABfPXEQayp28+J7OoFKRDqHThf0Zwzvw8AeOUx+bTnt8aYrIiKplsw9Y7PMbKaZvRvcLvDHTfT5jpktNLN5ZvaymQ1ssK7OzOYGj2dSvQH7K5ZmXH3CIJ1AJSKdRjJH9HuACe5+FDAKmGhm4xr1eQcodfcjgceBnzdYt9vdRwWP82kHLjy6iLzsOH9+XSdQiUj0tRj0nrAzeBoPHt6ozzR333sm0ttAUUqrTLGcjHSuGDeQlxZuZOXmXWGXIyLSqpIaozezmJnNBTYBU919xj66Xw083+B5lpmVmdnbZvbZfXzGpKBfWXl5eTJlHZQrxw8knpbGFF3sTEQiLqmgd/c6dx9F4kh9rJmNbKqfmV0OlAK/aNA8MLgz+ReB35jZ4GY+Y7K7l7p7aWFh4f5swwHp1TWLz47ux2Oz1+gOVCISafs168bdtwLTgImN15nZ6cC/A+e7+54Gr1kX/FwOvAKMPvByU+urJx5CVU0997+1MuxSRERaTTKzbgrNLD9YzgbOABY36jMa+BOJkN/UoL27mWUGyz2B44GFKav+IA3t3ZXTh/XivjdXsru6LuxyRERaRTJH9H2BaWY2D5hFYoz+WTO7zcz2zqL5BdAFeKzRNMphQJmZvUviL4GfuXu7CXqAr508mC2VNTw2W5dFEJFosvZ40lBpaamXlZW1yWe5O1+4803Kd+5h2k2nkB7rdOeQiUgEmNns4PvQT+n0qWZmfO3kwayp2K37yopIJHX6oAc4Y1hvDumZy59eW6bLIohI5CjoSdxXdtJJh7Bg3XZeX7I57HJERFJKQR/43Jj+9O6WyZ2vLAu7FBGRlFLQBzLTY3z1xEN4a/lHvLNaFzsTkehQ0Ddw6dgB5GXH+aOO6kUkQhT0DeRmpnPV+BKmLtzIBxt3hF2OiEhKKOgb+fL4ErLjMe7SUb2IRISCvpHuuRlcOnYAT7/7IWsqKlt+gYhIO6egb8JXTxpEzIw7X9VRvYh0fAr6JvTNy+bC0iIeL1vLhm1VYZcjInJQFPTN+PrJg6lzZ/Jrut2giHRsCvpmFBfkcMGofjw0cxWbd+5p+QUiIu2Ugn4fvnHKEPbU1nOPbjcoIh2Ygn4fhvTqwjlH9OX+t1axrbIm7HJERA6Igr4F3zx1CDv31DLlDR3Vi0jHpKBvwbC+3ThrRG/+8sYKtu3WUb2IdDwK+iRcf9pQdlTVcu8bK8MuRURkvyVzc/AsM5tpZu+a2Xtm9uMm+mSa2aNmttTMZphZSYN13w/a3zezs1Jcf5sY0S+PM4b3Zsr05Wyv0lG9iHQsyRzR7wEmuPtRwChgopmNa9TnamCLuw8Bfg38N4CZDQcuAUYAE4E/mlksRbW3qRtOG8r2qlru01G9iHQwLQa9J+wMnsaDR+P77V0A3BcsPw6cZmYWtD/i7nvcfQWwFBibksrb2Mj+eZw+rBd3T1/BDh3Vi0gHktQYvZnFzGwusAmY6u4zGnXpD6wBcPdaYBvQo2F7YG3Q1tRnTDKzMjMrKy8v36+NaCvXnzaUbbtrNFYvIh1KUkHv7nXuPgooAsaa2chUF+Luk9291N1LCwsLU/32KXFkUT6nD+vNn1/XWL2IdBz7NevG3bcC00iMtze0DigGMLN0IA/4qGF7oCho67BuPD0xVq+zZUWko0hm1k2hmeUHy9nAGcDiRt2eAa4Kli8E/unuHrRfEszKGQQMBWamqPZQjOyfx8QRfZjy+gqdLSsiHUIyR/R9gWlmNg+YRWKM/lkzu83Mzg/6TAF6mNlS4DvAzQDu/h7wV2Ah8AJwnbvXpXoj2tqNZwxlx55a7p6uK1uKSPtniQPv9qW0tNTLysrCLmOfrntoDq8s3sT0702ge25G2OWISCdnZrPdvbSpdToz9gDdeNpQKmvquOs13YVKRNo3Bf0BGtq7K58b1Z/73lzJpu26C5WItF8K+oNw4+mHUlvn/GHa0rBLERFploL+IAzokcPFxxTz0MzVrN1SGXY5IiJNUtAfpG9NGIKZccfLS8IuRUSkSQr6g9Q3L5srxg3k8dlrWVa+s+UXiIi0MQV9CnzjlMFkx2Pc/tIHYZciIvIpCvoU6NElk2tOPIS/z1/PvLVbwy5HROQTFPQpcs2JgyjIzeDnL7wfdikiIp+goE+Rrllxrjt1CNOXbmb6ks1hlyMi8jEFfQpdduwA+udn8/MXF9MeLy0hIp2Tgj6FsuIxbjx9KPPWbuO5+RvCLkdEBFDQp9znxxRxaO8u/OLFxVTX1oddjoiIgj7VYmnGzWcfzsqPKnl45uqwyxERUdC3hlMP68W4Qwq44+UlupG4iIROQd8KzIzvnz2Mj3ZVM/k13ZxERMKloG8lRxXnc+6Rfbn79RVs1GWMRSRECvpW9G9nHUZtfb0ujSAioUrm5uDFZjbNzBaa2XtmdkMTff7NzOYGjwVmVmdmBcG6lWY2P1jXvu8PmGIDe+RyxbgS/jp7DYvWbw+7HBHppJI5oq8FbnL34cA44DozG96wg7v/wt1Hufso4PvAq+5e0aDLqcH6Ju9nGGXXnzaEbllxfvrcIp1EJSKhaDHo3X29u88JlncAi4D++3jJpcDDqSmv48vPyeBbE4bw+pLNvPJBedjliEgntF9j9GZWAowGZjSzPgeYCDzRoNmBl8xstplN2sd7TzKzMjMrKy+PViBeeVwJA3vk8NO/L6K2TidRiUjbSjrozawLiQC/0d2bG3A+D3ij0bDNCe4+BjibxLDPSU290N0nu3upu5cWFhYmW1aHkJGexs0TD2fJpp08WrYm7HJEpJNJKujNLE4i5B909yf30fUSGg3buPu64Ocm4G/A2AMrtWObOLIPY0sKuP2lD9iuk6hEpA0lM+vGgCnAIne/fR/98oCTgacbtOWaWde9y8CZwIKDLbojMjN+eO5wKiqr+f0/l4Zdjoh0Iskc0R8PXAFMaDCF8hwzu9bMrm3Q73PAS+6+q0Fbb2C6mb0LzAT+7u4vpKz6DuaIojwuHFPEX95YwYrNu1p+gYhIClh7nPJXWlrqZWXRnHK/aXsVp/7yFcYP6cmfr+x0s01FpJWY2ezmprDrzNg21qtbFt84dQhTF27kjaW6E5WItD4FfQiuPmEQRd2z+fH/vqfpliLS6hT0IciKx/jBZ4bzwcadPPD2qrDLEZGIU9CH5KwRvTlhSE9un/oBH+3cE3Y5IhJhCvqQmBm3njecyuo6fqmrW4pIK1LQh2ho765cNb6ER2atZsG6bWGXIyIRpaAP2Q2nD6VHbgY/enoB9fXtb6qriHR8CvqQdcuK872JhzNn9VaemLM27HJEJIIU9O3AF8YUcfTA7vzs+cVsq9R1cEQktRT07UBamnHbBSPYUlnNr6a+H3Y5IhIxCvp2YkS/PK48roQH3l6lL2ZFJKUU9O3It884lILcDH6oL2ZFJIUU9O1IXnacW84Zxjurt/LILN2gRERSQ0HfznxudH/GHVLAz55fxGadMSsiKaCgb2fMjP/87BHsrqnjp88tCrscEYkABX07NKRXF7520mCenLOON5fpUsYicnAU9O3UNycMobggmx88tYA9tXVhlyMiHVgy94wtNrNpZrbQzN4zsxua6HOKmW1rcKvBHzVYN9HM3jezpWZ2c6o3IKqy4jH+87NHsLx8F3+ctizsckSkA0vmiL4WuMndhwPjgOvMbHgT/V5391HB4zYAM4sBfwDOBoYDlzbzWmnCyYcWcsGoftz5yjKWbtoRdjki0kG1GPTuvt7d5wTLO4BFQP8k338ssNTdl7t7NfAIcMGBFtsZ/fDc4WRnxLjlSc2tF5EDs19j9GZWAowGZjSx+jgze9fMnjezEUFbf6DhhPC1NPNLwswmmVmZmZWVl5fvT1mR1rNLJv9+zjBmrqzg0TLNrReR/Zd00JtZF+AJ4EZ3395o9RxgoLsfBfwOeGp/C3H3ye5e6u6lhYWF+/vySLuotIhxhxTw0+cWsXF7VdjliEgHk1TQm1mcRMg/6O5PNl7v7tvdfWew/BwQN7OewDqguEHXoqBN9oOZ8V+fP5Lq2np++NQC3DWEIyLJS2bWjQFTgEXufnszffoE/TCzscH7fgTMAoaa2SAzywAuAZ5JVfGdyaCeuXznjEN5aeFGnl+wIexyRKQDSU+iz/HAFcB8M5sbtN0CDABw97uAC4Gvm1ktsBu4xBOHnbVm9k3gRSAG3OPu76V2EzqPq08YxLPz1vOjpxcwfnAP8nMywi5JRDoAa4/DAKWlpV5WVhZ2Ge3Swg+3c/7vp3P+qH7cfvGosMsRkXbCzGa7e2lT63RmbAczvF83vn5K4vII/1y8MexyRKQDUNB3QN+cMITDenfl+0/O160HRaRFCvoOKDM9xi8vOorNO6u57dmFYZcjIu2cgr6DOqIoj6+fPJgn5qzVEI6I7JOCvgP71mmJIZybn5jP1srqsMsRkXZKQd+BZabH+NXFR1Gxq5pbn9GsVRFpmoK+gxvZP49vTRjK03M/5O/z1oddjoi0Qwr6CPjGqYM5qiiPHzw1n026Fo6INKKgj4B4LI1fXTyKyuo6bn5yvq6FIyKfoKCPiCG9uvC9iYfzz8WbeGjm6rDLEZF2REEfIV8aX8KJQ3vyk2cXsqx8Z9jliEg7oaCPkLQ045cXHUV2PMaNj8ylurY+7JJEpB1Q0EdM725Z/Nfnj2T+um385h8fhF2OiLQDCvoImjiyD/+ntJg7X13GW8s+CrscEQmZgj6ifnTecAb1yOXGR9+hYpfOmhXpzBT0EZWbmc4dl45my64avvv4u5pyKdKJKegjbGT/PG4++3D+sWgT9725MuxyRCQkCvqI+/LxJUw4vBc/fW4xC9ZtC7scEQlBMjcHLzazaWa20MzeM7MbmuhzmZnNM7P5ZvammR3VYN3KoH2umen+gG3MLDHlsiA3g28+NIcdVbpRiUhnk8wRfS1wk7sPB8YB15nZ8EZ9VgAnu/sRwE+AyY3Wn+ruo5q7n6G0roLcDH73xdGs2bJbl0gQ6YRaDHp3X+/uc4LlHcAioH+jPm+6+5bg6dtAUaoLlYNzTEkB//fMw/j7vPU88PaqsMsRkTa0X2P0ZlYCjAZm7KPb1cDzDZ478JKZzTazSft470lmVmZmZeXl5ftTliTpaycdwqmHFfKTZxcxb+3WsMsRkTaSdNCbWRfgCeBGd9/eTJ9TSQT99xo0n+DuY4CzSQz7nNTUa919sruXuntpYWFh0hsgyUtLM26/eBSFXTP5+gNz2KL59SKdQlJBb2ZxEiH/oLs/2UyfI4G7gQvc/ePTMd19XfBzE/A3YOzBFi0HrntuBndePobyHXu44dG51NVrvF4k6pKZdWPAFGCRu9/eTJ8BwJPAFe7+QYP2XDPruncZOBNYkIrC5cAdWZTPjy8YwWsflPNbXQ9HJPLSk+hzPHAFMN/M5gZttwADANz9LuBHQA/gj4nfC9QGM2x6A38L2tKBh9z9hVRugByYS44p5p3VW7jjn0s5siif04f3DrskEWkl1h6n2pWWlnpZmabct7aqmjouuustVmzexVPXHc+QXl3CLklEDpCZzW5uCrvOjO3EsuIx/nTF0WTF05h0fxnbdutkKpEoUtB3cv3ys/njZUezuqKSGx95R1/OikSQgl4YO6iA/zh/BNPeL+fnLy4OuxwRSbFkvoyVTuDycQNZvGE7f3p1OUMKu3BRaXHYJYlIiuiIXj5263kjOGFIT27523xmLNedqUSiQkEvH4vH0vjDZWMoLsjhaw/MZuXmXWGXJCIpoKCXT8jLjnPPVccA8JV7Z+kyCSIRoKCXTynpmcufryxl7dbdfPX+Mqpq6sIuSUQOgoJemnRMSQG/vngUZau2cNNf36Ve0y5FOizNupFmfebIvny4dRj/77lF9MnL4gefGUZwOQsR6UAU9LJP15w4iHVbdzNl+goKu2Zy7cmDwy5JRPaTgl72ycz40bnDqdhVzc+eX0xBTgYXH6M59iIdiYJeWpSWlrjB+JbKam5+ch55OXHOGtEn7LJEJEn6MlaSkpGexl2XH82RRfl866F3eH2Jbvco0lEo6CVpuZnp3PvlYzikMJev3l/GzBUVYZckIklQ0Mt+yc/J4IFrjqVffjZfuXcWc9dsDbskEWmBgl72W88umTx0zTi658a5csoM5q3dGnZJIrIPydwzttjMppnZQjN7z8xuaKKPmdkdZrbUzOaZ2ZgG664ysyXB46pUb4CEo09eFg9dM45u2XEuv1thL9KeJXNEXwvc5O7DgXHAdWY2vFGfs4GhwWMScCeAmRUAtwLHAmOBW82se4pql5AVF+Tw8FcV9iLtXYtB7+7r3X1OsLwDWAT0b9TtAuB+T3gbyDezvsBZwFR3r3D3LcBUYGJKt0BC1TDsL/vzDGav0he0Iu3Nfo3Rm1kJMBqY0WhVf2BNg+drg7bm2iVCigtyePRrx9GzayaX3z2T6Us2h12SiDSQdNCbWRfgCeBGd9+e6kLMbJKZlZlZWXm55mh3NP3zs3n0a+MY2COHr9w7i6kLN4ZdkogEkgp6M4uTCPkH3f3JJrqsAxqeF18UtDXX/inuPtndS929tLCwMJmypJ3p1TWLRyaNY1i/blz7wGz+Wram5ReJSKtLZtaNAVOARe5+ezPdngGuDGbfjAO2uft64EXgTDPrHnwJe2bQJhGVn5PBg9ccy/jBPfju4/P4w7SluOsSxyJhSuZaN8cDVwDzzWxu0HYLMADA3e8CngPOAZYClcCXg3UVZvYTYFbwutvcXd/WRVyXzHSmXHUM3338XX7x4vts2l7Fj84bQSxNlzgWCUOLQe/u04F9/gv1xCHbdc2suwe454Cqkw4rIz2N2y8eRa9uWUx+bTlrt+zmt5eOpkumrqMn0tZ0Zqy0mrQ045ZzhvGTz47klQ/Kueiut1i/bXfYZYl0Ogp6aXVXjBvIlKtKWVNRyQW/f4M5q7eEXZJIp6KglzZxymG9eOLr48mMp3HJn97m0Vmrwy5JpNNQ0EubOaxPV/73mydw7CEFfO+J+fzgqflU19aHXZZI5CnopU3l52Twly8dw6STDuGBt1dz0Z/eYk1FZdhliUSagl7aXHosjVvOGcadl41h+aadnPu76fxDZ9KKtBoFvYTm7CP68uz1J1BckM0195fxH8+8R1VNXdhliUSOgl5CNbBHLo9fO54vH1/CvW+u5ILfv8HiDSm/lJJIp6agl9BlxWPcet4I7vvKWCoqqzn/d29w16vLqK3TF7UiqaCgl3bj5EMLeeGGE5lweC9+9vxivnDXWyzZuCPsskQ6PAW9tCs9umRy5+Vj+P0XR7OmopLP3DGd3/5jCXtqNXYvcqAU9NLumBnnHtmPl759EmeN7MOv//EBZ//mdd5cqhuaiBwIBb20Wz27ZPK7S0dz31fGUlvvfPHuGXzr4XdYt1XXyxHZHwp6afdOPrSQl759EtefNpSX3tvAab96hV9P/YDK6tqwSxPpEBT00iFkxWN854xDefmmkzltWG9++/ISTvnFKzzw9ipqNDtHZJ8U9NKhFHXP4Q9fHMPj1x7HgIIcfvDUAs789Ws8PXcddfW6k5VIUxT00iGVlhTw2LXHcfeVpWTE0rjhkbmc+etXeeodBb5IY9Ye7+dZWlrqZWVlYZchHUR9vfP8gg3c8fIS3t+4g4E9crjmhEFceHQx2RmxsMsTaRNmNtvdS5tc11LQm9k9wLnAJncf2cT6fwMuC56mA8OAwuB+sSuBHUAdUNtcEY0p6OVA1Nc7Ly3cyF2vLmPumq10z4lz+biBXHbsQPrkZYVdnkirOtigPwnYCdzfVNA36nse8G13nxA8XwmUuvt+TYBW0MvBcHdmrdzC5NeW8fLiTaSZcdaI3lw+biDjBvUgTTcplwjaV9Anc3Pw18ysJMnPuhR4eD9qE0k5M2PsoALGDipg9UeVPDhjFY/MWsNz8zcwoCCHi0uL+MLRRfTNyw67VJE2kdQYfRD0z+7riN7McoC1wBB3rwjaVgBbAAf+5O6T9/H6ScAkgAEDBhy9atWq/dgMkX2rqqnj+QXreXTWGt5eXoEZjBvUgwtG9ePskX3Jy4mHXaLIQTmooZvgDUpoOej/D3C5u5/XoK2/u68zs17AVOBb7v5aS5+noRtpTas+2sXf3lnH03M/ZMXmXcRjxvjBPZk4sg9nDO9Nzy6ZYZcost/aKuj/Bjzm7g81s/4/gJ3u/suWPk9BL23B3Zm3dhvPzV/P8ws2sLqiEjMYVZzPhMN6cerhvRjet5vG9KVDaPWgN7M8YAVQ7O67grZcIM3ddwTLU4Hb3P2Flj5PQS9tzd1ZtH4HLy3cwLTFm3h37TYACnIzGD+4BycM6cmxh/SgpEcOZgp+aX8O6stYM3sYOAXoaWZrgVuBOIC73xV0+xzw0t6QD/QG/hb8o0gHHkom5EXCYGYM79eN4f26cePph1K+Yw+vLyln+tLNvLF0M8/OWw9AYddMxpYUMHpAPqMHdGdEv25kxTVXX9o3nTAl0gJ3Z1n5LmauqGDWygpmrqj4+Aqa8ZhxWJ+ujOyXx4h+3RjWtxuH9elK1yx9uStt66CHbtqagl7au03bq3hnzVbeWb2V9z7cxvx129haWfPx+v752Qzt3YUhhV0Y3KsLg3rmMqhnLr26ZmroR1rFQQ3diMin9eqWxVkj+nDWiD5A4qj/w21VLF6/ncUbdrB4ww6WbtrJW8s+Yk/tv66umR2PUVyQTVH3HIq7Z9MvP5u++dn0y8uid7csenXLJDNdQ0GSWgp6kRQwM/rnZ9M/P5vThvX+uL2+3lm3dTcrP9rFyo8qWbl5F2sqKlm7ZTezVlawo+rT19TPz4lT2CWTnl0y6dk1kx65GXTPyaAgN05+Tgb5OXHyshOPbllxumalkx7T9QmleQp6kVaUlmYUF+RQXJDDiUM/vX5HVQ0btlXx4bYqNm6rYtOOKjZu38OmHVVU7KpmwbptbN65p8lfCA3lZMTIzUyna2Y6uZnpHz/PzoiRE4+RnREjOx4jMx4jMz2NrOBnZnoamfEYGbE0MtKNjFiMeMyIp6eREUsjPWakp6URjxmxtMRy4qeRFvyMpRlptvcnGppqhxT0IiHqmhWna1acob277rNfTV09Wytr2FJZzbbdNWytrGHb7hp2VNWwfXctO6pq2FVdy849deysqqGyuo7yHXvYtaeWqpo6dtfUUVld94lhpNaSZpBmiV8Ee5djZhAs296fQOJ3wt5fEGBY8PNfvzDM9vb71/q99i5a8H6f0Mzvm+Z+DaXqF9TBvEv3nAz+eu1xKamjIQW9SAcQj6VR2DWTwq4Hd9auu7Ontp6qmjqqa+vZU1vPntrEL4CaOqe6tp7aunqq6+oTy/VOTV09tXVOXb1T505tvVNbV09dvVMfPK+vd+rqoc4d93/1xfl4ee+8j/pgud4dh6Ddqa8HJ7Fub7sTPGFv278mj/jH2/Sv5Ybb2eT2N/sf5gD+Yzb5Ngf3Rt1aabaWgl6kEzEzsuIxzf3vZPQNjohIxCnoRUQiTkEvIhJxCnoRkYhT0IuIRJyCXkQk4hT0IiIRp6AXEYm4dnmZYjMrBw707uA9gc0pLKcj6IzbDJ1zuzvjNkPn3O793eaB7l7Y1Ip2GfQHw8zKmrsmc1R1xm2GzrndnXGboXNudyq3WUM3IiIRp6AXEYm4KAb95LALCEFn3GbonNvdGbcZOud2p2ybIzdGLyIinxTFI3oREWlAQS8iEnGRCXozm2hm75vZUjO7Oex6WouZFZvZNDNbaGbvmdkNQXuBmU01syXBz+5h15pqZhYzs3fM7Nng+SAzmxHs80fNLCPsGlPNzPLN7HEzW2xmi8zsuKjvazP7dvD/9gIze9jMsqK4r83sHjPbZGYLGrQ1uW8t4Y5g++eZ2Zj9+axIBL2ZxYA/AGcDw4FLzWx4uFW1mlrgJncfDowDrgu29WbgZXcfCrwcPI+aG4BFDZ7/N/Brdx8CbAGuDqWq1vVb4AV3Pxw4isT2R3Zfm1l/4Hqg1N1HAjHgEqK5r+8FJjZqa27fng0MDR6TgDv354MiEfTAWGCpuy9392rgEeCCkGtqFe6+3t3nBMs7SPzD709ie+8Lut0HfDaUAluJmRUBnwHuDp4bMAF4POgSxW3OA04CpgC4e7W7byXi+5rELU6zzSwdyAHWE8F97e6vARWNmpvbtxcA93vC20C+mfVN9rOiEvT9gTUNnq8N2iLNzEqA0cAMoLe7rw9WbQB6h1VXK/kN8F2gPnjeA9jq7rXB8yju80FAOfCXYMjqbjPLJcL72t3XAb8EVpMI+G3AbKK/r/dqbt8eVMZFJeg7HTPrAjwB3Oju2xuu88Sc2cjMmzWzc4FN7j477FraWDowBrjT3UcDu2g0TBPBfd2dxNHrIKAfkMunhzc6hVTu26gE/TqguMHzoqAtkswsTiLkH3T3J4PmjXv/lAt+bgqrvlZwPHC+ma0kMSw3gcTYdX7w5z1Ec5+vBda6+4zg+eMkgj/K+/p0YIW7l7t7DfAkif0f9X29V3P79qAyLipBPwsYGnwzn0Hiy5tnQq6pVQRj01OARe5+e4NVzwBXBctXAU+3dW2txd2/7+5F7l5CYt/+090vA6YBFwbdIrXNAO6+AVhjZocFTacBC4nwviYxZDPOzHKC/9f3bnOk93UDze3bZ4Arg9k344BtDYZ4WubukXgA5wAfAMuAfw+7nlbczhNI/Dk3D5gbPM4hMWb9MrAE+AdQEHatrbT9pwDPBsuHADOBpcBjQGbY9bXC9o4CyoL9/RTQPer7GvgxsBhYAPwPkBnFfQ08TOJ7iBoSf71d3dy+BYzEzMJlwHwSs5KS/ixdAkFEJOKiMnQjIiLNUNCLiEScgl5EJOIU9CIiEaegFxGJOAW9iEjEKehFRCLu/wNprwkwnVv4GAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 定义一个简单的网络\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.fc1 = nn.Linear(1, 10)\n",
    "        self.fc2 = nn.Linear(10, 1)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = self.fc1(x)\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "# 随机生成训练数据\n",
    "x = torch.randn(100, 1)\n",
    "y = x.pow(2) + 0.1 * torch.randn(100, 1)\n",
    "\n",
    "# 实例化网络\n",
    "net = Net()\n",
    "\n",
    "# 定义损失函数和优化器\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = torch.optim.Adadelta(net.parameters())\n",
    "\n",
    "# 记录训练损失\n",
    "losses = []\n",
    "\n",
    "# 开始训练\n",
    "for epoch in range(100):\n",
    "    # 前向传播 + 反向传播 + 优化\n",
    "    output = net(x)\n",
    "    loss = criterion(output, y)\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 记录损失\n",
    "    losses.append(loss.item())\n",
    "\n",
    "# 绘制训练损失图\n",
    "plt.plot(losses)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在训练过程中，每隔一个epoch记录一次损失，最后绘制损失的变化图。可以看到随着训练的进行，损失逐渐减少。这说明网络的权重在逐渐拟合训练数据。\n",
    "\n",
    "**梗直哥提示：这个例子只是为了给出一个大致的框架，帮助你理解如何使用Adadelta算法来训练一个网络。你可以根据自己的需要来修改代码，例如调整网络的结构、使用不同的数据集等。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Next 6-10 Adam算法](./6-10%20Adam算法.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
